System for Low-Latency Animation of Talking Heads

ABSTRACT

Methods and apparatus for rendering a talking head on a client device are disclosed. The client device has a client cache capable of storing audio/visual data associated with rendering the talking head. The method comprises storing sentences in a client cache of a client device that relate to bridging delays in a dialog, storing sentence templates to be used in dialogs, generating a talking head response to a user inquiry from the client device, and determining whether sentences or stored templates stored in the client cache relate to the talking head response. If the stored sentences or stored templates relate to the talking head response, the method comprises instructing the client device to use the appropriate stored sentence or template from the client cache to render at least a part of the talking head response and transmitting a portion of the talking head response not stored in the client cache, if any, to the client device to render a complete talking head response. If the client cache has no stored data associated with the talking head response, the method comprises transmitting the talking head response to be rendered on the client device.

PRIORITY INFORMATION

The present application is a continuation of U.S. patent applicationSer. No. 11/778,228, filed Jul. 16, 2007, which is a continuation ofU.S. patent application Ser. No. 10/423,789, filed Apr. 25, 2003, nowU.S. Pat. No. 7,260,539, the contents of which are incorporated hereinby reference in their entirety. The present application is also relatedto International Application No. PCT/US2004/012402, filed Apr. 22, 2004,the contents of which is incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to network-based talking heads and morespecifically relates to an architecture to reduce the latency of talkinghead animation in a network environment.

2. Introduction

A growing number of websites use natural language interfaces tocommunicate with their customers, to guide customers for more successfulself-service and to enhance the user experience. In some cases, somesites provide varying images of talking heads to express moods (happy,offended, sad) in addition to the text displayed in the browser window.In the progression of this technology, animated talking faces forcustomer service and sales applications on the Internet further enhancethe communication between an organization and its customers.

The use of natural language interfaces in web-based interactionstypically consists of several major components and steps: (1) the clientuses a regular web browser such as Internet Explorer or Netscape; (2)the user types text into a text box on a web page; (3) this text is sentto the server; (4) the server transmits the text to a dialog managerwhich consists of several modules including natural languageunderstanding, dialog control and natural language generation; and (5)the dialog manager transmits responsive text to the server whichforwards the text with the appropriate web page(s) to the client.Compared to simple websites that serve up web pages without furtherprocessing at the server, the latency of the server response asperceived by the client is increased by the response time of the dialogmanager.

In cases where the user interaction with the website further includes atalking face, two additional steps must occur: (1) speech needs to besynthesized using a speech synthesizer (TTS); and (2) based on thephonemes created by the TTS, a renderer animates the face. While speechsynthesis can be done faster than realtime, the latency of a TTS system(Time to first audio) usually exceeds 0.3 seconds. In web interactions,people are often exposed to considerable latencies due to slow downloadspeeds; but as web interactions become more like face-to-faceconversations, low latencies are essential. Delays above 0.3 seconds inresponse are noticeable and irritate the user.

Based on the phoneme and related information from the TTS, the talkinghead is animated. While face animation can be done in real time, theface renderer also adds latency to the system. Depending on the facemodel, time to first video can exceed 0.5 seconds. High quality faceanimation systems use coarticulation models to compute the mouth shapes.The current mouth shape depends on previous sounds. Furthermore, themouth moves in anticipation of sounds adding further to the latency ofthe face animation. The current invention solves these problems byintroducing caches at the client and server side that can presenttalking head animations with a low latency while the server isgenerating new parts of the animation.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates an example architecture according to an embodiment ofthe invention; and

FIG. 2 illustrates a method according to another embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

Latency of face animation impedes communications between a web serverand a user, thus making the user experience unnatural. Disclosed hereinis an architecture that reduces the latency of the face animation systemusing a cache on the client and/or a cache on the server. The serverkeeps track of the information in the client cache. Furthermore, theserver-based TTS and visual TTS (VTTS) systems may use a cache in orderto avoid computing identical animations for multiple users. Thearchitecture is suitable to support servers that use a dialog manager tocreate dynamic contents as well as servers that send static content tothe client.

Caching on the Internet follows to a large extent the same principles ascaching in computers. The web pages that have been downloaded previouslyare stored in a local cache. If this page is accessed within a limitedperiod of time, the cached page is shown instead of downloading a newpage from the server. Extensions of this principle are proxy caches,where the most popular pages are cached on multiple servers, so thatpages can be retrieved from a server that is ‘closer’ to the user thanthe original server.

The present invention relates to a caching system and method thatreduces the latency in network-based dialogs between a user and atalking head. Dialogs are particularly demanding with respect to latencysince users expect a swift reaction to their input. Caching can reducelatency and required bandwidth any time there is a repetition in thedata/animations presented to the viewer or when there is a predictablereaction. Fortunately, in dialogs repetitions and predictable reactionsare very common, making caching mechanisms particularly effective.

Caching can reduce latency and required bandwidth by storing animations,graphics and data at the client in multiple ways. In contrast toconventional caching on the Internet, dialog-caching requires storingpart of the computer responses on the client prior to their playouttime. Actual computer responses consist then of a mixture of informationpreemptively cached on the client and new information sent from theserver. This information needs to be presented without the user noticingfrom which source it comes. This preemptive caching can be static in thesense that it is the same or dynamic where the dialog manager controlsthe caching based on dialog probabilities. Examples of a static cachinginclude phrases such as “Good morning,” and “Glad to have you back.”Dynamic caching relate to information that changes such as a checkingaccount amount or the day or time. Phrases like “Hello, your accountbalance is $130.23” will have a static portion (the words in thesentence) and a dynamic portion (the account amount).

The data stored in either a client cache and/or a server cache may beparameters related to audio information, text information, or visualinformation. The information may be audio or visual parameters or actualaudio signals or video images. The Audio information may refer tospeech, text, or mark-up information that accompanies text used forspeech.

FIG. 1 shows an exemplary architecture 100 that comprises componentssuch as a server 102 communicating with a dialog manager (DM) 104. TheDM 104 communicates with a text-to-speech (TTS) module 106 and a visualTTS (VTTS) module 108. The TTS module 106 communicates with an audiocodec 110 and VTTS 108 communicates with a visual codec 112. The DM 104,audio codec 110 and visual codec 112 communicate with the server 102.

The server 102 communicates with a client 114 by receiving HTTP requestsand transmitting audio and visual information to the client 114 togenerate a talking head and web pages to carry on a dialog with a user.An audio/visual cache 116 communicates with the client 114. A renderer120 and a web browser 118 communicate with the client 114. The servermay also have a database or server cache 117 that stores informationrelated to the client cache 117. The information may be a copy of theclient cache 116 or parameters identifying what information is stored inthe client cache 116. For example, the server may store information suchas sentences 1-12 and 25-45 are stored in the client cache 116. Theserver cache 117 may also store dynamic information that will beinserted into numerous static/dynamic phrases to be spoken by thetalking head. For example, the server could calculate that today is Feb.12, 2003, and will only need to do this once for all the conversationsoccurring that require this dynamic information. In this regard, you canstore audio and visual information on the server cache 117 and draw uponthis information in at least one dialog or multiple dialogs where theaudio and visual data is required.

The server 102 and client 114 communicate over the Internet using theappropriate and known protocols. To begin a description of an embodimentof the invention, assume the server 102 receives an HTTP request fromthe client 114. In the following example steps, illustrated in FIG. 2,it is assumed that the server 102 needs to use the DM 104 in order toserve the request from the client 114.

The server 102 transmits text to the DM 104 (202). This text may be theinput from the client 114 extracted from the HTTP request or some textderived from the web page associated with the requested URL. The DM 104analyzes the text and formulates a response considering the state of thedialog, the click trail of the user and other items. (204). Thetext-based response may contain markup that indicates emotional stateslike happiness, disgust, anticipation, etc. The text-based response issent to the TTS module 106 as well as to the server 102 (206). If theserver 102 determines that the text-based response is stored in theclient cache 116, the server 102 sends a signal to the client 114 toplay the response from the cache 116 (208). Furthermore, the server 102sends the appropriate web page to the client 114 (210). If the server102 determines that the current response is not available at the clientcache 116, it instructs the TTS 106 and VTTS system 108 to provide theresponse (212). The TTS 106 synthesizes the speech and a list ofphonemes, durations, tags and the markup of the text for the faceanimation system and transmits the speech and other data, if any, to theaudio codec 110 (214). The audio codec 110 compresses the audio andpackages the data such that the server can serve it to the client (216).The face animation server (VTTS) 108 computes animation parameters thatwill enable the client to render the face (218). Animation parametersmay include head and eye motion, facial expressions, videos of the mouthand/or eye area of the face model and videos of the entire head.

The visual codec 112 compresses the output of the VTTS 108 and packagesthe data such that the server 102 can serve it to the client 114 (220).The server 102 sends the audio and visual data to the client 114 (222).The data includes timestamps that enable the decoder to synchronouslypresent the rendered face model and the associated audio. The server 102may send a flag to the client 114 instructing the client to store thisinformation in its cache 116. The server 102 keeps track of the clientcache 116 either by keeping a copy of the client cache 116 or by keepingonly the response of the dialog manager 104 in the server cache (notshown) as an indication of contents in the client cache 116.Furthermore, the server 102 may choose to send a web page that providesnew or updated information to the user. The web page may be sent priorto the transmission of the audio/visual data.

The terminal device of the user (not shown) includes the client 114 thatinterprets the received data. The HTTP data is sent to the browser 118.If the server 102 sends a flag, audio/visual data is sent to therenderer 120. Using another flag, the server 102 may indicate that thisaudio/visual data is to be stored in the database on the client 114.Alternatively, a flag can indicate that the audio/visual data is to besent to both the renderer 120 as well as the database 116. As usedherein, the terms database and cache typically have a similar meaning.Alternatively, the server might request that data from the client cache116 is sent to the renderer 120.

Upon receipt of the data from the server 102, the client 114 decodes theaudio/visual data, renders the face model based on the visual data andpresents it synchronized with the audio. Furthermore, the web pagereceived from the server 102 is rendered on the browser 118 of theterminal.

Compared to an architecture without a cache on a client, the latency ofthe system is only reduced if the audio visual data is available in thelocal cache. Therefore, the server 102 according to an aspect of theinvention begins storing sentences for bridging delays in the cache 116of the client as soon as the client 114 connects to the server 102 andwhile waiting for input from the client 114. Typical sentences to bestored are greetings (“Hello”, “Good-bye”) and sentences for maskingdelays (“Please wait a second”, “just a moment please”, “Let me checkour database”). The server 102 may also store templates for sentenceslike “The time is now . . . .” In this example, the server 102 wouldonly have to send the variable part of the template—the actual time like“5 pm” to the client 114. While the template is played, the variablepart is received from the server 102, resulting in a very low latencyresponse to the user.

Other information to be stored in the client cache 116 is idle behaviorof the face model. This information may be termed “behavioraltemplates”. Such templates may provide talking head visual informationsuch as listening behavior and waiting behavior. Such behavior templatesprovide a variety of visual prosody patterns that make the interactionwith the user more natural given the context. The behavioral templatescan cover visual prosody for any component of a conversation. Inaddition to the listening behavior discussed above, the talking behaviorcan also be included. For example, if the talking head needs to ask aquestion twice, the visual prosody may change to be more serious andintense. The visual prosody may be different for greeting a user with“Good morning!” than informing the user that “you owe $55 in late fees.”Further, there may be transition visual prosody that enables the talkinghead to move appropriately if the system determines that the user isabout to finish a phrase and it is the turn of the talking head tospeak. A behavioral template to simulate a normal conversation can coverany visual queue or motion of the talking head.

The client cache 116 size might be limited and therefore the client 114in one aspect of the invention informs the server 102 when it removesinformation from the cache 116. Alternatively, the server 102 informsthe client 116 which parts are not needed anymore.

Many parts of a dialog are predetermined. An indication of this is thata dialog system like AT&T's “How May I Help You?” uses only recordedprompts to interact with its users. Therefore, the server 102 canpre-compute many sentences of a dialog. In one aspect of the invention,the system stores these pre-computed sentences in a server cache (notshown). Every time the dialog manager 104 sends a response with asynthesis request to TTS 106 and VTTS 108, the server cache is checked.If response is in the cache, it is forwarded immediately to the server102, thus not increasing latency compared to a text-only web-baseddialog system. Requests that need to be synthesized are stored in theserver cache for potential future use.

In order to leverage the multitude of dialogs that happen between aserver 102 and its many clients 114, the server cache of audio/visualdata may be common to all dialogs. In this case, the server 102 willonly have to synthesize responses containing dynamic information(personal data like phone numbers, volatile information like time).Computation time may be further reduced by pre-computing sentences usingtemplates. Templates are commonly used in dialog systems. For example,in a shopping environment, users might want to know the value of theitems in their shopping cart. The dialog manager 104 might be able toconvey the total price using sentences like “Your total is $X,” or“Today, you ordered for $X.” The audio and visual part of the fixed partof the template may be pre-computed. Together with the pre-computedaudio/visual data, the TTS 106 and VTTS 108 will save the state of theiralgorithms such that they can continue processing a response as soon asthe dynamic part is known.

Embodiments within the scope of the present invention may also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer. By way of example, and notlimitation, such computer-readable media can comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to carryor store desired program code means in the form of computer-executableinstructions or data structures. When information is transferred orprovided over a network or another communication connection (eitherhardwired, wireless, or combination thereof) to a computer, the computerproperly views the connection as a computer-readable medium. Thus, anysuch connection is properly termed a computer-readable medium.Combinations of the above should also be included within the scope ofthe computer-readable media.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, components, and data structures, etc. that perform particulartasks or implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of theinvention may be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunication network. In a distributed computing environment, programmodules may be located in both local and remote memory storage devices.

Although the above description may contain specific details, they shouldnot be construed as limiting the claims in any way. Other configurationsof the described embodiments of the invention are part of the scope ofthis invention. For example, a client device could receive via adownloading process a dialog manager as well as a TTS module. In thisregard, although above the preferred embodiment shows a dialog managerand TTS module (and other modules) remote from the client device 114, itis contemplated that the location on the network of the TTS module anddialog manager are irrelevant to the present invention. Accordingly,these various modules may exist on any node of any network and theparticular claims are not limited to any architecture in this regard.The appended claims and their legal equivalents should only define theinvention, rather than any specific examples given.

1. A computer-readable medium storing instructions which, when executedby a computing device, cause the computing device to render a talkingentity, the instructions comprising: A. generating a talking entityresponse to a user utterance; B. retrieving cached sentences associatedwith bridging delays in a dialog and cached templates for use withdialogs; C. if at least one sentence or template relates to the talkingentity response:
 1. using the at least one sentence or template torender at least a part of the talking entity response; and
 2. receivinga portion of the response not stored in a cache at the computing deviceto render a complete talking entity response; D. if none of thesentences or templates relates to the talking entity response: 1.receiving the talking entity response to be rendered; and E. renderingthe talking entity response on the computing device.
 2. Thecomputer-readable medium of claim 1, wherein the instructions furthercomprise storing idle behavior associated with the talking entity. 3.The computer-readable medium of claim 2, wherein the idle behaviorrelates to one or more of listening behavior and waiting behavior of thetalking entity.
 4. The computer-readable medium of claim 1, wherein thecomputing device comprises two or more separate devices whichcommunicate one with another.
 5. The computer-readable medium of claim4, wherein the two or more separate devices comprises a server deviceand a client device.
 6. The computer-readable medium of claim 1, whereinthe talking entity is a talking head.
 7. The computer-readable medium ofclaim 1, wherein at least one cached template is associated with abehavior template for the talking entity when the talking entity isasking a question for the second time.
 8. The computer-readable mediumof claim 1, wherein at least one cached template is associated withlistening behavior.
 9. The computer-readable medium of claim 1, whereinat least one cached template is associated with behavior when waitingfor an answer to a question.
 10. A method of rendering a talking entityon a computing device having a client cache capable of storingaudio/visual data associated with rendering the talking entity, themethod causing a computing device to perform steps comprising: A.generating a talking entity response to a user utterance; B. retrievingcached sentences associated with bridging delays in a dialog and cachedtemplates for use with dialogs; C. if at least one sentence or templaterelates to the talking entity response:
 1. using the at least onesentence or template to render at least a part of the talking entityresponse; and
 2. receiving a portion of the response not stored in acache at the computing device to render a complete talking entityresponse; D. if none of the sentences or templates relates to thetalking entity response:
 1. receiving the talking entity response to berendered; and E. rendering the talking entity response on the computingdevice.
 11. The computer-implemented method of claim 10, furthercomprise storing idle behavior associated with the talking entity. 12.The computer-implemented method of claim 11, wherein the idle behaviorrelates to listening behavior and waiting behavior of the talkingentity.
 13. The computer-implemented method of claim 10, wherein thecomputing device comprises two or more separate devices whichcommunicate one with another.
 14. The computer-implemented method ofclaim 13, wherein the two or more separate devices comprise a serverdevice and a client device.
 15. The computer-implemented method of claim10, wherein the talking entity is a talking head.
 16. Thecomputer-implemented method of claim 10, wherein at least one cachedtemplate is associated with a behavior template for the talking entitywhen the talking entity is asking a question for the second time. 17.The computer-implemented method of claim 10, wherein at least one cachedtemplate is associated with listening behavior.
 18. A system forrendering a talking entity, the system having a processor, amachine-readable storage medium storing functional modules, and a cachestoring data associated with rendering the talking entity, the computingdevice comprising: A. a module controlling the processor to generate atalking entity response to a user utterance; B. a module controlling theprocessor to retrieve cached sentences associated with bridging delaysin a dialog and cached templates for use with dialogs; C. if at leastone sentence or template relates to the talking entity response:
 1. amodule controlling the processor to use the at least one sentence ortemplate to render at least a part of the talking entity response; and2. a module controlling the processor to receive a portion of theresponse not stored in a cache at the computing device to render acomplete talking entity response; D. if none of the sentences ortemplates relates to the talking entity response:
 1. a modulecontrolling the processor to receive the talking entity response to berendered; and E. a module controlling the processor to render thetalking entity response on the computing device.
 19. The computingdevice of claim 18, further comprising a module controlling theprocessor to store idle behavior associated with the talking entity. 20.The computing device of claim 19, wherein the idle behavior relates tolistening behavior and waiting behavior of the talking entity.